Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review

Machine Learning and Artificial Intelligence (AI) more broadly have great immediate future potential for transforming almost all aspects of medicine. However, in many applications, even outside medicine, a lack transparency AI applications has become increasingly problematic. This is particularly pronounced where users need to interpret the output systems. Explainable (XAI) provides rationale that allows understand why system produced given output. The can then be interpreted within context. One area XAI Clinical Decision Support Systems (CDSSs). These systems support medical practitioners their clinic decision-making absence explainability may lead issues under or over-reliance. Providing explanations how recommendations are arrived at will allow make nuanced, some cases, life-saving decisions. CDSS, field general, amplified by ethical fair fact trained with historical data reinforcement agent actions biases should uncovered. We performed systematic literature review work to-date application CDSS. Tabular processing XAI-enabled most common, while CDSS text analysis least common literature. There interest developers provision local explanations, there was balance between post-hoc ante-hoc as well model-specific model-agnostic techniques. Studies reported benefits use such it could enhance decision confidence clinicians, generate hypothesis about causality, which ultimately leads increased trustworthiness acceptability its incorporation clinical workflow. we found an overall distinct context and, particular, user studies exploring needs clinicians. propose guidelines implementation explore opportunities, challenges, research needs.